Method for the transmission of image data 

Field of the invention 

The present invention concerns a method for the transmission of 
image data as well as devices suitable therefor. 

5 Related Art 

Image data, in particular digital data that can be represented as 
images to a user with suitable reproduction means, are often compressed 
BQ before transmission in order to reduce the transmission times and then 

\y\ decompressed before or during reproduction. Different standards for the 

W 10 compression and decompression of image data, for example the different 
" f MPEG (Moving Pictures Expert Group) standards, have already been 

V 

O described. 
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if! Patent application W098/3331 5 (University of Texas) describes 

another image data compression method that is based on the physiological 
15 characteristics of the human eye, which needs a maximal resolution only in 
the area of the image projected onto the eye's fovea (viewing area). Margi- 
nal image areas can be reproduced with a lower resolution without noti- 
ceable loss of quality. This patent application thus proposes to continuously 
determine the image area viewed by the user and to use the maximal reso- 
20 lution only for this image area, whilst a lower resolution is used for further 
removed areas. The viewed image area typically constitutes about two 
percent of the entire image. The image strongly compressed in this manner 
is then transmitted over a transmission channel between two computers. 

This method is suitable in particular for the transmission of 
25 images that are destined to being reproduced on a small display, for 
example on the screen of an office computer. When the viewing point is 
able to move quickly, for example with very wide image formats or when 
the pupil moves jerkily, it can happen that the reaction time of the system 
is too slow, so that image areas with a poor resolution can suddenly appear 
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at the center of the viewed area. Furthermore, this method is not suitable 
for the simultaneous sending of image data to a plurality of users. 

It is an aim of the present invention to propose a new and impro- 
ved method for the compression and transmission of images, in particular a 
5 method with which the aforementioned disadvantages can be avoided. 

Brief Summary of the Invention 

u> According to the present invention, this aim is achieved in parti- 

cular through the elements of the independent claims. Further advan- 
tageous embodiments are moreover described in the dependent claims and 
!U 10 in the description. 
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In particular, these aims of the invention are achieved with a 
method for the transmission and reproduction of image data in which 



- said image data can be transmitted by a sending device to at 
least one communication terminal and reproduced by image 

III 15 reproducing means in said at least one communication 

terminal, the current viewing direction of the communication 
terminal's user being determined, 

- first image data corresponding to the entire image area are 
transmitted with a low resolution over a first transmission 

20 channel, 

- said current viewing direction being sent to said sending 
device over a reverse channel, 
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second image data corresponding to the image area currently 
viewed by the user are transmitted with a higher resolution 
over a second transmission channel, and 



- said first and second image data are superimposed and 
simultaneously reproduced. 

This has the advantage that the first image data are transmitted 
over a first transmission channel (for example a broadcast channel), whilst 
only the second image data, corresponding to the currently viewed image 
area, are transmitted over a second transmission channel (for example a 
costly bi-directional mobile radio network). Thus, a higher resolution than 
that in the aforementioned patent application W098/33315 can be used for 
the marginal areas, without the data quantity transmitted over the payable 
mobile radio network being increased. 

Through the broadcast channel, the same image data can be sent 
with a low resolution to all users, whilst image data corresponding to the 
image area viewed by the individual users can be sent personally with a 
higher resolution to every user over the second transmission channel. 

Description of the drawings 

Hereafter, an embodiment of the present invention will be 
described by means of an example. The example of the embodiment will be 
illustrated by the following attached drawings, in which: 

Fig. 1 shows a block diagram of the system according to the 

invention. 

Fig. 2 shows diagrammatically the segmentation of the bitmap. 

Fig. 3 shows the resolution requirement determined by the eye's 
physiology when image data is viewed by several users. 

Detailed Description of the Invention 

In Figure 1, the reference number 1 refers to a sending device 
that can consist for example of a commercially available communication 



server having hardware and software components for communicating over 
two different transmission channels 4, 7 with a plurality of communication 
terminals 8. The reference number 10 refers to image data, for example 
digital data files, whose contents can be represented with suitable repro- 
5 duction means to a user 9 as images, for example as motionless or prefe- 
rably animated images. The image data can for example correspond to a 
digitized image sequence (for example a movie or a television program) or 
other multimedia data, for example HTML data containing fixed and/or 
animated image data and/or sound data. In a variant embodiment of the 
io invention, the image data 10 in the particularly interesting image area and 
Cf ' image details onto which the viewer will with a high probability direct his 

I|l eye, are specially marked, as will be explained further below. 
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ifjj The sequence of image data is read sequentially through a 

SI sequencer (not represented) in the sending device 1, the sequence being 

* 15 optionally controlled by the user 9, for example in the case of image data 

pj with hypertext control elements. The sequencer can for example consist of 

[|| a hardware and/or software module that reads the images of the image 

sequence and forwards them to an (optional) segmentation module 2. 
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The segmentation module 2 partitions the image in different 
20 segments, as represented in Figure 2. For this, it uses indications A about 
the current viewing direction of the individual user 9, for example the 
coordinates x, y of all points 22 onto which the eye direction of at least one 
user 9 is currently directed. Preferably, one or several displacement vectors 
23 corresponding to the speed of the currently viewed points 22, are 
25 additionally received or determined in the segmentation module. 

The segmentation module 2 determines on the basis of these 
indications A the area or areas 21 that are currently viewed by at least one 
user 9. As will be described further below, these areas are compressed with 
a higher resolution and transmitted to the corresponding user over the 
30 transmission channel 7. The entire image, including the marginal areas 25, 
is compressed with a lower resolution with another algorithm and sent to 
all users over the transmission channel 4. The size of each area 21 can 



preferably be adapted to the bandwidth of the transmission channel 7. 
Typically, the size of each high resolution area 21 corresponds to about 2 to 
4 percent of the entire image. 

In a preferred embodiment, the segmentation module 2 further- 
more tries to determine in advance one or several viewing areas 24. To this 
effect, the segmentation module can for example take into account the eye 
movements indicated with the displacement vector 23, the movement of 
the viewed object or of the person on the image and/or specially marked 
details in the image data, for example hyperlinks in the case of multimedia 
image data. The segmentation module 2 can for example store successive 
positions of the eyes and/or of the viewed object on the image and 
determine in advance the next expected viewing direction, for example by 
means of suitable regression functions, it being possible to mark manually 
certain often viewed areas. 

In the variant embodiment represented in Figure 2, the different 
image areas 21, 24, 25 are rectangular. They can however also have another 
shape and different sizes and overlap partially or totally. 

The image areas 25 removed from the viewing point are sent to a 
first encoding module 3 where they are compressed for example with a 
MPEG algorithm or, in the case of motionless images, with a JPEG or GIF 
algorithm. The first encoding module 3 uses preferably spatial, temporal 
and chrominance compression methods. Furthermore, the first encoding 
module preferably conducts an error encoding and a channel encoding. 

The image areas encoded by the encoding module 3 are then 
sent from a sender 4 in broadcast mode and can thus be received by a 
plurality of receivers 8 by means of individual broadcast receivers 80. The 
sender 4 can for example send the data as DAB (Digital Audio Broad- 
casting) or DVB (Digital Video Broadcasting) image data, as television data, 
for example digitally encoded television data, or as TCP-IP data over the 
Internet, etc. The data prepared by the encoding module 3 can even be 
copied on magnetic and/or optical data carriers, for example on CD or on 
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DVD, and in this manner can be distributed commercially and purchased by 
a plurality of users. 

The image data received by the first receiver 80 in the 
communication terminal 8 are copied in a first cache memory 81, image 
5 areas 25 from several successive individual images being preferably stored. 
In this manner, variable transmission times can be compensated through 
the two transmission channels 4 and 7. 
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u The data selected by the segmentation module from the image 

£3 areas 21 and 24 viewed currently and in future are forwarded to a second 

jjj 10 encoding module 5 that compresses each area 21, 24 viewed currently and 
HJ in future with a foveal compression function, i.e. with a function that 

y supplies as a result image data with a higher resolution at close range to 

SI the viewed point 22 than in the further removed image areas, as this is 

jLj described in the aforementioned patent application W098/33315. In a 

111 15 preferred embodiment of the invention, at least certain parameters of the 
foveal function can be dependent on the bandwidth of the second 
transmission channel. Furthermore, the second encoding module 5 
fU preferably conducts an error encoding and a channel encoding. 

Each image area 21 encoded by the second encoding module 5 is 
20 sent over a second transmission channel 7 to all users 9 who are currently 
viewing this image area and can thus be received by these users by means 
of second receivers 83 (for example mobile radio terminals) in the 
communication terminals 8. The second transmission channel 7 is a bi- 
directional communication channel that allows near real-time connections, 
25 for example a publicly connected telephone network, for example a digital 
mobile radio network, for example according to GSM or UMTS, or a fixed- 
line network, for example according to ISDN, or a TCP-IP network, for 
example Internet. 
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A variant embodiment of the invention (not represented) forgoes 
the segmentation module 2 and all the individual images 10 are forwarded 
to the two encoding modules 3 and 5. The first encoding module 3 can 
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consist in this case of a standardized module that compresses all areas 21, 
24 and 25 of all individual images. The second encoding module 5 can 
compress all the images according to a foveal function. 

The communication terminals 8 are preferably personally 
portable, electrically autonomous and are preferably made as pocket 
devices. In the embodiment in which the telecommunication network 7 is a 
mobile radio network, at least certain of the communication terminals 8 
are mobile radio devices, for example mobile radio telephones or 
communication-capable laptops or palmtop computers that comprise also a 
broadcast receiver 80. Such combined communication terminals have 
already been described for example in patent applications WO99/60712 and 
WO99/60713 (both in the name of Swisscom AG). The mobile radio device 
83 can for example exchange data over the mobile radio network 7 with 
the aid of GPRS (Generalized Packet Radio Service), or according to a 
suitable protocol in the service channel. The first receiver, for example a 
DVB receiver, and the second receiver 83, for example a UMTS terminal, are 
preferably combined in a single unit (for example in a single housing), but 
can in variant embodiments be integrated in several units combined with 
one another. In the latter case, the two units are preferably connected over 
a contactless interface, for example according to RrdA, Bluetooth or 
HomeRF. 

The image data received by the second receiver 83 in the 
communication terminal 8 are copied in a second cache memory 84, several 
successive image areas 21 resp. 24 being preferably stored in the second 
25 cache memory 84. 

A superimposition module 82 in the communication terminal 8 
reads the image data in both cache memories 81 and 84 and crossfades 
them. To this effect, each image area 21, 24, 25 in both cache memories 
preferably carries a number indicating to which individual image it belongs, 
30 in order to compensate different transmission times through the two 
transmission channels 4 and 7. The image area 21, 24 selected by the 
module 82 in the second cache memory 84 is selected according to the 
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currently viewed image area. If the cache memory is large enough, it can 
also be used to pause, restart and/or wind forwards and backwards the 
reading of the stored image data through appropriate user commands. 

The image signals corresponding to the image data combined by 
5 the module 82 can then be reproduced through a display device 85 of the 
communication terminal 8. In a preferred embodiment of the invention, 
the display device consists of a so-called Virtual Retina Display (VRD), that 
projects the image signals onto the retina of the eye of the user 9 of the 
fsA communication terminal 8. Such VRD devices have been described in patent 

O io applications WO 94/09472 and WO 97/37339. The display device 85 can be 

GO 
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Si communication terminal 8 can be implemented in common or separate 

units, it being possible to connect the display device 85 in a first unit with 
il{ 15 components of the second unit for example over a wire-bound or over a 
W wireless interface. 

m 



jpj supplied with image data in different formats over a suitable interface. 

The display device 85 and the further components of the 
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\\l As represented diagrammatically in Figure 1, the communication 

terminal 8 comprises a viewing direction measuring module 86 that can 
determine the viewing direction x, y of the user 9. Such a viewing-direction 

20 measuring-module (Eye Tracking System ETS) has also been described in the 
aforementioned patent application WO 94/09472. The indication A of the 
viewing direction x, y is forwarded to the module 82 which can, as has 
already been mentioned, determine which image area 21, 24 from the 
second cache memory 84 is to be read. Thereafter, this indication A is 

25 forwarded to the transceiver 83, linked through a module (not represe- 
nted) with user-specific indications and forwarded in real-time from the 
transceiver 83 over the reverse channel of the second transmission channel 
7 to the sending device 1. According to the type of the second transmission 
channel, these indications A can be transmitted for example as USSD 

30 message or preferably over the service or data channel. 



In one embodiment, the viewing direction is preferably measured 
by means of a measuring module mounted on spectacles opposite the 
user's head and measuring preferably the position of the pupil of at least 
one eye. 

In another embodiment, the viewing direction is preferably 
measured by means of a measuring module mounted on spectacles 
opposite the user's head and measuring the position and/or orientation of 
the head of the user 9 in comparison with a reference world by means of a 
measuring system, preferably an inertial system mounted on the user's 
head.' Thus, image data that are larger than the user's field of vision and 
the contents of which can move when the user turns his head can be 
projected to the user 9. In this way, the current viewing direction 22 can be 
linked with the valid viewed area 21. With this method, films with very 
large angle views can be represented to a user. 

Said additional user-specific indications that are sent back 
similarly to the viewing direction A, preferably comprise a user 
identification stored for example in a personal identification module 830 
(for example a SIM card). Furthermore, these additional indications can 
include other commands from the user 9 (for example a URL selected by the 
user in the case where the represented image contains hyperlinks) or other 
selections in menus. These additional indications can be transmitted 
together or separately. At least certain additional indications can be 
encrypted and/or signed electronically. 

In the sending device 1, the received indication A of the currently 
viewed point x, y is used by the segmentation module 2 to determine the 
currently viewed image area 21 and, if necessary, to predetermine the 
future image area 24 and to forward these image areas 21, 24 to the 
encoding module 5. 

The user identification transmitted as additional indication can 
be used by a billing center 6 on the sender's side that can bill the viewing 
of the image data. The billed amount can for example be debited directly 
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from a prepaid money amount stored on the identification module 830, 
from a bank account from a credit card or through an invoice (for example 
as part of the telephone bill) and can be billed for example per time unit of 
the received image data, per title, per additional information or pages 
5 and/or in combination with a subscription. User-specific settings for the 
invoice (for example the preferred billing address and type of invoice) can 
be stored in the billing center in a user data bank 80. The billing center can 
be integrated in the sending device 1 or, in a variant embodiment, can be 
administered by another institute (for example by the operator of the 
10 network 7). 

is** 

1XDI 

m Selection commands and instructions that are entered by the user 

ill 9 of the telecommunication terminal 8 and transmitted as additional 

Fj indications over the second transmission channel 7 are received and 

yj 

Sf processed further by said sequencer (not represented), so that for example 

3: 
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15 image data requested by the user, for example subtitles, HTML or WML 
j{j pages, sound sequences etc. can be obtained and transmitted over both 

W ' transmission channels to the communication terminal 4. 

m 

ill This invention is particularly suited for the simultaneous 

transmission of image data 1 (for example movies or television programs) 

20 to a plurality of users 9, common data being sent over a broadcast channel 
4 whilst data that are dependent on the personal viewing direction are 
transmitted over an addressed channel 7. It can however also be used for 
video conferences or for telephone calls with image transmission between 
two partners. In this way, the transmission of the image data, whilst taking 

25 into account the resolution requirements determined by the eye's 

physiology, is tailored to the predetermined, maximal bandwidth of the 
communication channels 4 and 7. 

Although several details of this invention's description relate to 
the special case of an embodiment in a GSM mobile radio network, the one 
30 skilled in the art will understand that this method can also be used with 
other types of mobile and fixed-line networks, for example with AMPS, 
TDMA, CDMA, TACS, PDC, HSCSD, GPRS, EDGE or UMTS mobile radio 
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networks, in particular with WAP (Wireless Application Protocol)-capable 
mobile radio networks. This invention can furthermore be used in other 
networks, in particular the Internet. 



